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ABSTRACT 


This  report  is  a summary  of  the  activities  and  technical  reports  for  the 
Energy  Model  Validation  Procedure  Development  project  undertaken  by  the 
Operations  Research  Division  for  the  Department  of  Energy,  using  DOE’s  Midterm 
Oil  and  Gas  Supply  Modeling  System  (MOGSM)  as  a test  vehicle.  The  reports 
cover:  (1)  assessment  of  the  documentation  of  MOGSM;  (2)  analysis  of  (a)  the 

model  methodology,  (b)  characteristics  of  the  input  and  other  supporting  data, 
(c)  statistical  procedures  undergirding  construction  of  the  model,  and  (d) 
sensitivity  of  the  outputs  to  variations  in  input;  as  well  as  (3)  guidelines 
and  recommendations  for  the  role  of  these  in  model  building  and  developing 
procedures  for  their  evaluation. 

Key  Words:  Assessment;  documentation;  energy;  forecasting;  mathematical 

models;  sensitivity  analysis. 


I.  INTRODUCTION 


This  report  summarizes  the  activities  of  a DOE-sponsored  project  for  "Energy 
Model  Validation  Procedure  Development".  The  project* s basic  purpose  was  to 
develop  and  apply  standards  and  procedures  for  the  assessment  of  analysis 
systems  (models)  utilized  by  the  Energy  Information  Agency  of  the  Department 
of  Energy.  This  work,  and  other  related  DOE  activities,  has  the  goal  of 
developing  methods  for  finding  the  degree  of  confidence  in  each  system’s 
results  and  the  circumstances  under  which  a system  may  be  used. 

To  initiate  this  research,  the  DOE  Midterm  Oil  and  Gas  Supply  Modeling  System 
(MOGSM)  was  selected  as  a test  vehicle  for  idea  development  and  experimen- 
tation. Any  proposed  assessment  standards  and  procedures  were  to  be  applied 
to  the  latest  version  of  MOGSM.  As  given  In  the  project  statement  of  work, 
objectives  of  this  project  were:  (1)  to  develop  methods  useful  for  validating 

EIA  analysis  systems;  (2)  to  establish  a team  of  analysts  consisting  of  NBS 
personnel  and  outside  consultants  that  will  accumulate  and  maintain  expertise 
in  validation  procedures  as  applied  to  DOE  analysis  systems;  (3)  to  apply  pro- 
posed validation  procedures  to  the  DOE  midterm  oil  and  gas  supply  models;  and 
(4)  to  specify  system  validation  standards  and  procedures  based  on  the  experi- 
ences of  the  midterm  oil  and  gas  supply  model  evaluation. 

These  objectives  were  to  be  accomplished  by  a set  of  tasks  that  included  an 
evaluation  of  the  documentation  and  the  identification  of  documentation  defi- 
ciencies; evaluation  of  model  attributes,  including  data,  mathematical  formu- 
lation, statistical  and  other  estimation  procedures;  analysis  of  system  sensi- 
tivity, system  performance  and  computer-related  system  characteristics;  and 
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the  specification  of  assessment  activities  necessary  to  support  the  determina- 
tion of  confidence  in  a system’s  results. 

It  should  be  stressed  that  assessment  of  the  MOGSM  was  not  our  primary  goal. 
NBS’s  task,  as  described  in  the  statement  of  work,  was  to  use  the  MOGSM  as  a 
means  for  developing  an  assessment  methodology  for  possible  use  by  DOE  and 
other  energy  modeling  groups.  Model  assessment  as  a formal  discipline  is  In 
its  early  stages  of  development.  This  project  was  designed  to  contribute  to 
this  new  and  important  "modeling"  activity. 

The  body  of  this  report  consists  of  condensations  of  the  reports  produced  in 
the  course  of  execution  of  the  project  tasks,  followed  by  a summary  of  overall 
conclusions  and  recommendations . Thus  the  report  could  be  viewed  as  an 
executive  summary  for  the  project. 
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II.  PROJECT  REPORTS 


The  activities  associated  with  the  development  of  energy  model  validation  pro- 
cedures are  described  in  detail  in  the  10  reports  listed  below  by  title;  com- 
plete references  are  given  in  Section  V.  The  following  subsections  of  this 
section  discuss  each  report  according  to  the  scheme  given  below,  and  each  sub- 
section includes  a brief  statement  of  the  conclusions  and  recommendations  re- 
sulting from  the  assessment  activities  described  in  the  listed  reports. 


Section  A; 

Interim  Report  on  Model  Assessment  Methodology:  Documentation 

Assessment. 


Section  B: 

An  Annotated  Restatement  of  the  Midterm  Oil  and  Gas  Supply  Modeling 
System  Methodology. 


Section  C: 

Investigation  of  Underlying  Data:  Midterm  Oil  and  Gas  Supply 

Modeling  System. 

Section  D: 

System  Sensitivity  and  Stability  I:  Model  Validation,  Simulation, 

and  Sensitivity  Analysis. 

System  Sensitivity  and  Stability  II:  A Statistical  Method  for  the 

Assessment  of  Model  Sensitivity  to  Input  Variables. 


Data  Extrapolation  and  Statistical  Forecasting. 

The  Role  of  the  Statistician  in  Energy  Model  Development  and  Vali- 
dation. 


Section  E: 

Sensitivity  Analysis  of  DOE  Forecasts  of  Midterm  Oil  and  Gas  Supply 
for  the  1978  Annual  Report  to  Congress. 


Section  F; 

Concepts  of  Model  Confidence. 


Section  G; 

Conversion  Costs:  A Practical  Exercise  in  Model  Portability. 
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A.  Interim  Report  on  Model  Assessment  Methodology:  Documentation  Assessment ^ 

We  approached  the  task  of  evaluating  the  MOGSM  documentation  from  the  view- 
point of  model  assessors  seeking  answers  to  the  following  questions: 

1.  What  was  the  model  supposed  to  be? 

(Documentation  accompanying  the  model  is  the  only  proper  source  of 
such  information.) 

2.  What  did  the  model  turn  out  to  be? 

(The  computer  code  is  a necessary,  but  not  sufficient  source  for 
this  information.) 

3.  Is  the  resulting  form  consistent  with  the  intent? 

We  proposed  to  answer  these  questions  by  obtaining  a sufficient  understanding 
of  the.  conceptual  model  and  computer  code  to  enable  us  to  run  the  computer 
system  under  different  test  scenarios.  WTe  discovered  early  that  the  HOGSM 
documents  available  from  DOE  did  not  contain  sufficient  information  to  pursue 
this  course  directly.  The  MBS  project  staff  was  able  to  augment  the  MOGSM 
documentation  by  locating  a number  of  ancillary  documents.  With  the  support 
of  the  original  model  developers  and  DOE  staff,  we  were  then  able  to  develop 
an  understanding  of  the  conceptual  model  and  its  computer  realization  suffi- 
cient to  permit  continuation  of  the  project.  Without  this  very  time  consuming 
and  painstaking  effort,  the  project  could  not  have  continued. 

Our  initial  approach  to  assessing  the  available  MOGSM  documents  was  to  examine 
them  in  terras  of  previously  proposed  model  documentation  guidelines^  to 

*See  reference  [1]. 

^The  guidelines  used  were  as  presented  in  Gass,  S.  I.,  "Computer  Model  Docu- 
mentation: A Review  and  An  Approach,"  NB5  Special  Publication  500-39, 

National  Bureau  of  Standards,  Washington,  D.  C. , February  1979. 
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determine  whether  documents  that  conform  to  these  proposed  guidelines  were 
sufficient  for  the  needs  of  model  assessors.  However,  the  information  in  the 
DOE  documentation  was  not  close  enough  to  the  information  requirements  of  the 
guidelines  for  this  comparison  to  be  accomplished.  We  therefore  developed  a 
revised  set  of  model  documentation  guidelines,  based  on  a review  of  the  MOGSM 
documentation.  The  project  report  presents  the  revised  guidelines  and 
discusses  (a)  the  denciencies  of  the  available  MOGSM  documents  and  (b)  how 
MOGSM  documents  that  conformed  to  these  guidelines  would  have  benefited  DOE. 


The  documentation  guidelines  proposed  in  the  report  represent  a hierarchial 

approach  to  information  requirements  in  which  the  four  categories  of  document 

types  are  associated  with  phases  or  levels  of  model  use.  A higher  level 

includes  all  of  the  documents  of  a lower  level.  The  levels  and  brief 

descriptions  of  the  required  documents  follow. 

0 

Level  I:  Rote  Operation  of  the  Model 

Level  I is  concerned  with  requirements  for  rote  execution  of  computer  runs,  i. 
e.  the  "ground  rules  for  setting  up  and  running  the  model"  on  a particular 
computer  and  for  verifying  the  correctness  of  the  execution.  The  document 
types  are: 


o Operations  Manual — provides  computer  operations  personnel  with  a 
description  of  the  software  and  operational  environment  so  that  the 
software  can  be  run; 

o Data  Base  Description:  Physical/Logical  Characteristics — presents  in- 

structions for  data  entry,  tape  and  file  labeling  conventions,  support 
software  descriptions,  logical  data  characteristics  (e.g.,  arrangement, 
relationships),  and  physical  characteristics  (e.g.,  storage,  access); 
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o Software  Description:  User  Level — describes  the  functions  performed 

by  the  software  in  non-ADP  terminology,  so  that  the  user  organization 
can  determine  how  to  apply  and  operate  the  model. 


Level  II:  Model  Use 

Level  II  involves  an  explanation  of  a given  set  of  scenarios  and  enables 
construction  of  new  scenarios  and  interpretation  of  output.  The  relevant 
documents  are  those  specified  in  Level  I and: 


o Mathematical  Description — describes  the  complete  details  of  the 

mathematical/ logical  model,  assumptions  and  hypotheses,  rationale  for 
the  form  of  the  model,  discussion  of  alternatives,  and  restrictions 
on  model  use; 

o Data  Requirements  Report:  Sources,  Transformations,  and  Justifica- 

tion— describes  the  detailed  data  needs  of  the  model  including  both 
input  variables  and  "hardwired"  parameters; 

o Process  Description — describes  the  underlying  physical,  economic, 
technological,  and  behavioral  processes  to  be  modeled  for  readers 
unfamiliar  with  the  topic  area. 


Level  III:  Model  Maintenance 


In  this  level,  the  documentation  addresses  modification  of  the  computer  code 
(and  perhaps,  therefore,  the  conceptual  model)  to  investigate  scenarios  which 
range  beyond  originally  conceived  limits  or  assumptions.  The  relevant  docu- 
ments are  those  specified  in  Levels  I and  II  and: 


o Software  Description:  Programmer  Level — provides  a maintenance  pro 

grammer  with  the  information  necessary  to  understand  the  programs, 
their  operating  environment,  and  their  maintenance  procedures; 

Maintenance  Log — records  changes  made  to  the  models  and/ or  its  data 
in  a manner  which  facilitates  the  extraction  of  an  audit  trail. 


o 
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Level  IV:  Model  Assessment: 

In  order  to  conduct  a third-party  assessment,  model  assessors  should  have  all 
model  documentation  available  to  them.  For  a variety  of  reasons,  the  report 
types  listed  below  may  not  have  been  produced,  but  if  they  do  exist  they 
should  be  made  available  to  the  assessment  team. 


The  relevant  documents  are  those  specified  in  Levels  I,  II,  and  III  and: 

o Assessment  Report — describes  any  model  assessment  plan  agreed  to  by  the 
user/sponsor  and  model  developer,  and  the  results  of  implementing  that 
plan; 

o Model  Application  Report — describes  for  decision  makers  the  results  of 
exercising  the  model  to  answer  specific  questions  or  to  study  the  be- 
havior of  the  problem  environment  as  represented  by  the  model; 

o Model  Summary — presents  a concise  nontechnical  description  of  the  model 
so  that  a broad  audience  can  determine  If  it  is  of  interest  to  them; 

o Historical  Record — describes  the  questions  or  problems  that  led  to  the 
decision  that  the  model  was  needed,  and  how  the  model  is  to  be  used  to 
address  these  issues; 

o all  other  documents  written  about  the  model  and  not  specifically 
listed  above. 


This  report  stresses  that  model  documentation  should  not  be  judged  on  the 
basis  of  whether  the  designated  documents  exist,  but  rather  by  whether  or  not 
all  of  the  information  required  in  each  document  is  readily  available  in  a 
well-organized  manner. 
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?>,e  commendations 

Specific  recommendations  to  DOE  are  the  following: 

o The  four-level  documentation  guidelines  should  be  adopted  provision- 
ally by  DOE  and  applied  to  future  modeling  activities. 

o Model  documentation  should  proceed  pari  passu  with  model  development 
to  improve  efficiency  of  production  and  quality  of  documentation. 

o To  improve  current  deficiencies  in  DOE  model  documentation,  the  re- 
port identifies  those  information  requirements  that  are  sufficiently  crucial 
to  continuing  model  use  and  necessary  for  model  assessment  to  warrant  post  hoc 
preparation. 

o A model  assessment  team  should  have  at  least  oae  member  who  has  no 
initial  knowledge  of  the  model. 

B.  An  Annotated  Restatement  of  the  Midterm  Oil  and  Gas  Supply  Modeling  System 

I 

Methodology^ 

The  NBS  project  to  develop  methodologies  for  the  assessment  of  energy  models 
began  with  an  assessment  of  the  MOGSM  documentation.  One  conclusion  of  the 
documentation  assessment  was  that  the  existing  documentation  did  not  contain 
information  sufficient  for  us  to  obtain  a thorough  understanding  of  the  mathe- 
matical/logical structure  of  the  model.  However,  through  many  discussions 
with  the  model  developers,  the  structure  of  the  model  was  exposed  and  the  mod- 
el assessment  effort  continued.  This  NBS  report  arose,  then,  through  our  re- 
cording for  our  own  use  of  our  interpretation  of  the  mathematical/logical 
structure  of  the  model.  It  was  pieced  together  from  portions  of  the  original 
documentation,  from  discussions  with  the  developers,  and  our  analysis. 

^See  reference  [9]. 
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Since  our  major  concern  was  the  development  of  assessment  methodology,  we  did 
not  intend  to  supplement  the  "primary"  documentation  of  the  model  we  were 
assessing.  However,  the  need  for  at  least  an  internal  document  became 
apparent.  As  the  production  of  this  document  progressed,  we  realized  that 
this  activity  was  advancing  our  general  methodology-development  goals.  It 
clarified  and  synthesized  many  of  our  notions  about  documentation  that  were 
incorporated  into  the  guidelines  reviewed  in  Section  A of  this  summary  report. 
We  also  believe  that  the  existence  of  this  report  will  help  advance  the  most 
important  by-products  of  proper  model  assessment — making  this  model  more 
"transparent",  i.  e. , more  open  to  peer  review  and  more  understandable  to 
policy  makers  who  rely  on  information  produced  by  models. 

Our  restatement  report  describes  the  methodology  used  by  the  Department  of  En- 
ergy (DOE)  to  estimate  oil  and  natural  gas  supply  curves  for  use  by  the  mid- 
term energy  forecast  system.  It  first  describes  the  key  features  of  the  oil 
and  natural  gas  supply  process  modeled  in  MOGSM.  It  next  describes,  in 
general  terms,  the  model  used  to  project  future  oil  supply  as  a function  of 
federal  and  corporate  policy  actions  and  market  prices  for  crude.  Finally,  it 
discusses  the  estimation  of  natural  gas  supply,  highlighting  the  differences 
between  it  and  the  oil  supply  estimation. 

A number  of  points  need  to  be  made  concerning  this  assessment  activity.  MOGSM 
consists  of  three  submodels:  the  Economic  Submodel,  the  Resource  Base  Sub- 

model, and  the  Drilling  Submodel.  Descriptions  of  both  the  Economic  Submodel 
and  the  Resource  Base  Submodel  are  based  on  the  verbal  descriptions  provided 
by  the  model  developers.  No  computer  code  listings  were  examined  to  see  if 
there  were  discrepancies  between  the  structure  described  by  the  modelers  and 
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the  computer  implementation.  We  have,  however,  requested  the  model  developers 
to  review  a draft  of  this  document  to  ensure  that  what  we  have  written  is  con- 
sistent with  their  "conceptual"  model. 

Information  on  the  Drilling  Submodel  was  obtained  by  reading  the  computer  code 
labeled  "OILDRL  78”.  This  exercise  uncovered  discrepancies  between  the  com- 
puter implementation  and  descriptions  in  either  the  existing  documentation  or 
in  verbal  statements  made  by  the  developers.  These  discrepancies  are  noted  in 
the  report. 

A major  portion  of  the  model  and  its  methodology  was  totally  undocumented  at 
the  outset  of  the  present  exercise.  Thus,  the  methodology  used  to  determine 
the  costs  of  exploratory  and  developmental  drilling  are  presented  for  the 
first  time  in  this  document,  thereby  allowing  review  of  that  submodel. 

It  is  a difficult  task  to  document  work  performed  by  others  and  to  record  ver- 
bal explanations  of  a highly  complex  model.  Furthermore,  we  do  not  believe 
that  the  task  is  complete  until  the  computer  code  of  both  the  Economic  Submod- 
el and  the  Resource  Base  Submodel  have  been  reviewed  by  a third  party  to  un- 
cover discrepancies  between  the  implementation  and  the  "conceptual"  model  doc- 


umented. 
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Examination  of  the  MOGSM  methodology  revealed  questionable  aspects  of  portions 
of  the  model;  tacit  assumptions  or  outright  discrepancies  between  stated  pro- 
cess relationships  and  computer  program  equations.  These  include  the  follow- 
ing: 


o The  time  order  of  discovery  of  oil  pools  is  strictly  related  to  size, 
i.e.,  probability  estimates  about  the  resource  base  are  translated 
into  deterministic  equations  in  the  model. 

o In  some  cases,  the  magnitudes  of  finding  rates  are  based  on  only  one  or 
two  data  points. 

o Exploratory  equipment  (rigs)  are  double  counted  in  projecting  drilling 
requirements. 

o Finding  rate  estimates  used  in  conjunction  with  the  factors  for 
ultimate  recovery,  known  as  Hubbert  multipliers,  result  in  double 
counting  of  some  oil  production. 

o Planned  and  realized  depreciation  are  determined  by  inconsistent 
calculations. 

o The  model  is  configured  to  treat  14  regions  in  some  calculations  but 
only  12  in  others. 

o For  offshore  regions  a constant  production  rate  is  assumed  in  the 
economic  submodel  while  a nonlinear  production  function  is  assumed 
within  the  resource  submodel. 


These  problems  have  been  brought  to  the  attention  of  the  model  developers,  re- 
sulting in  some  modifications  of  the  computer  programs  (which  have  not  been 
reassessed).  The  sensitivity  of  model  outputs  to  the  identified  anomalies  has 
not  yet  been  determined  completely,  although  some  of  the  sensitivity  analysis 
discussed  later  in  Section  E of  this  report  bears  on  the  finding  rate  question 
and  the  resource  base  estimates. 
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Recommendations 


Some  conclusions  and  recommendations  relating  to  assessment  methodology  deriv- 
ing from  the  preparation  of  this  report  are: 

o An  assessment  report  is  a useful  part  of  the  extended  documentation  of 
a model.  It  should  record  model  anomalies  or  deficiencies. 

o Assessment  documentation  must  conform  to  standards  required  of  the  rest 
of  model  documentation. 

o A description  of  a process  being  modeled  is  essential  for  model  assess- 
ment. Such  a description  must  exist  and  be  referenced  in  the  method- 
ology documentation.  It  serves  as  a safeguard  against  misinterpreta- 
tion in  the  application  of  a model. 

o The  only  sure  source  for  identification  of  discrepancies  between  a 
conceptual  and  a realized  model  is  the  computer  program.  Someone  must 
be  prepared  to  read  code  in  the  process  of  assessment.  Third-party 
assessment  of  computer  code  can  uncover  coding  inconsistencies  (unsus- 
pected by  the  model  builders)  in  different  portions  of  a model. 

o Assessment  findings  and  assessor-produced  additional  model  documenta- 
tion should  be  reviewed  by  the  model  developers.  If  possible,  the 
modelers'  responses  should  be  incorporated  into  the  assessment  report. 

o The  statement  that  the  process  of  documentation  can  uncover  weaknesses 
in  one's  understanding  of  the  mechanics  or  the  assumptions  of  a model 
applies  to  model  assessors  as  well  as  model  developers. 


C.  Investigation  of  Underlying  Data:  Midterm  Oil  and  Gas  Supply  Modeling 

System^- 

This  report  presents  the  findings  of  the  task  to  investigate,  using  existing 
data  documentation,  the  sufficiency  of  the  MOGSM's  underlying  data.  The  ob- 
jective of  the  study  was  to  define  and  achieve  a state  of  “audit  readiness" 
for  MOGSM's  data,  not  to  perform  a data  audit  or  validation  study.  A state  of 
"audit  readiness”  is  defined  to  be  that  state  of  an  analysis  system's  input 
data  and  source  data  which  permits  an  audit  on  that  data  without  further  as- 
sembly preparation,  or  collection.  A data  audit  is  a study  to  determine  the 


1-See  reference  [3]. 
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appropriateness,  accuracy,  timeliness,  and  completeness  of  the  data.  The 
study  was  achieved  by  activities  described  by  the  following  subtasks: 

1.  Summarize  DOE's  data  documentation  of  the  model's  input  data  (both 
internal  and  external  to  the  computer  programs  that  implement  the  model). 

2.  Provide  a complete  set  of  the  input  data  operated  on  by  the  model, 
including  data  elements  embedded  in  the  coding. 

3.  Provide  a complete  set  of  the  source  documents  containing  the  raw 
data  from  which  the  model's  input  data  are  derived. 

4.  Specify  the  intermediate  analysis  and  processing  steps  by  which  raw 
data  are  transformed  into  input  data. 

5.  Identify  possible  alternative  sources  of  raw  data. 

6.  List  errors  found  in  the  raw  data  or  the  input  data,  including  any 
discrepancies  between  the  purpose  for  which  the  raw  data  were  originally 
gathered  and  the  purpose  for  which  they  are  used  in  the  model.  (This  task  was 
specifically  not  intended  to  include  any  particularly  expensive  research 
effort  to  answer  this  question.) 

7.  Develop  a precis  of  the  input  data  and  data  documentation.  This  pre- 
cis is  to  enable  EIA's  Office  of  Energy  Information  Validation,  to  audit  the 


data 
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The  final  report  contains  three  major  sections:  the  data  precis , the  input 

data  listings,  and  the  summary  of  data  transformations.  The  presentation  of 
data  in  this  fashion  represents  a tentative  approach  for  describing  an  analy- 
sis system’s  input  data. 

Recommendations 

With  respect  to  a model's  data,  generic  findings  about  assessment  include: 

o The  data  associated  with  an  analysis  system  should  be  regarded  as  part 
of  that  system.  Specifically,  an  analysis  system  should  always  in- 
clude an  audit  of  the  data  associated  with  the  system, 
o The  assessment  performed  under  contract  to  NBS  included  a description 
of  the  method  used  to  achieve  "audit  readiness.”  Since  no  audit  took 
place,  a methodological  approach  for  auditing  large  data  sets  was  not 
developed.  This  next  stage  of  data  assessment  is  critical  and  should 
be  undertaken. 

o The  subsequent  process  of  auditing  an  analysis  system's  input  data  has 
the  two  distinct  components  of  addressing  the  source  data  and  the  data 
transformations.  These  components  may  require  different  skill  mixes 
in  the  audit  team. 

o An  analysis  system's  overall  design — including  its  data  structure  and 
implementing  programs — has  a significant  impact  on  the  effort  required 
for  assessment.  The  needs  of  model  assessment  should,  therefore,  be 
reflected  in  the  designs  of  future  systems. 
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The  report  also  presents  two  specific  conclusions  about  the  data  used  in 
MOGSM: 

o Within  the  limits  of  this  study,  we  found  no  errors  in  the  MOGSM  input 
data  set*  We  did,  however,  note  a number  of  inconsistencies  between 
original  purposes  of  certain  source  data  and  their  uses  in  MOGSM. 
o Many  of  MOGSM  data  elements,  especially  those  related  to  the  economics 
of  exploration  and  production,  come  from  a few  sources.  Many  have 
a weak  statistical  basis  and/or  rely  significantly  on  human  judgment. 
However,  we  did  not  locate  any  acceptable  alternatives  to  these 
sources. 

D.  Statistical  Aspects  of  Model  Assessment* 

Several  studies  were  carried  out  to  address  the  appropriateness  of  statistical 
estimation  methodologies  of  MOGSM  and  the  sensitivity  of  model  results  associ- 
ated  with  the  particular  parameter  values  specified  in  MOGSM.  One  of  our 
tasks  was  to  determine  the  adequacy  of  methods  used  to  derive  parameter  values 
contained  in  the  model’s  mathematical  representation.  Particular  emphasis  was 
to  be  placed  upon  an  examination  of  the  rationale  for  the  estimation  proce- 
dures selected  for  the  models  and  of  alternative  possibilities.  Another  task 
called  for  a sensitivity  assessment  of  the  model  by  comparison  of  outputs  gen- 
erated by  alternative  values  to  key  model  input  parameters.  Particular  empha- 
sis was  to  be  placed  upon  an  evaluation  of  model  sensitivity  and  stability 
relative  to  such  alternatives. 


*See  references  [3, 4, 5, 6]. 
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Appropriate  strategies  for  making  reliable  forecasts  from  historical  data  are 
not  known.  However,  there  are  in  fact  well-established  statistical  techniques 
for  projecting  data  trends  into  the  future.  These  usually  fall  into  the  sub- 
ject areas  of  "Forecasting"  and  "Time-Series  Analysis."  The  scope  of  this 
work  is  confined  to  what  might  be  called  short  and  mid-range  forecasting 
methods,  since  these  are  the  horizons  most  relevant  to  the  supply  models  under 
assessment. 

In  the  study  reports,  the  more  important  techniques  are  described  in  detail, 
in  addition  to  some  of  the  subjective  aspects  of  forecast  model  development. 
Included  are:  a brief  discussion  of  the  general  nature  of  such  models  and 

their  possible  application  in  energy  analysis;  a fairly  formal  definition  of 
the  basic  forecasting  problem;  the  major  alternative  methods;  performance 
criteria;  and  energy  applications.  A key  finding  here  is  that  the  technique 
for  calculating  the  statistical  uncertainty  in  the  supply  projections  of  the 
MOGSM  is  inadequate.  To  correct  this,  strategies  are  indicated  for  passing 
alternative  scenarios  through  the  models  in  a statistically  correct  and  more 
meaningful  manner.  The  major  established  technique  for  doing  this  is  called 
model  sampling.  The  complete  details  of  this  approach  are  provided,  along 
with  a discussion  of  its  potential  use  in  energy  modeling. 

Recommendations 

Based  on  the  statistical  and  sensitivity  analysis  tasks,  the  following  conclu- 
sions and  comments  are  offered.  Although  these  are  substantially  just  common 
sense,  they  have  been  emphasized  here  because  they  are  frequently  overlooked 
in  the  press  of  analysis  in  response  to  tight  deadlines. 
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o Statistical  assumptions  should  always  be  stated  clearly.  Their 

plausibility  and  their  relevance  to  the  requirements  of  the  statisti- 
cal methods  used  should  be  assessed.  For  example,  the  time  period 
from  which  the  statistical  relationship  is  derived  for  short-term 
forecasting  should  be  relevant  to  the  period  for  which  the  forecast  is 
being  made.  In  other  words,  forecasts  from  historical  data  embody  an 
assumption  that  the  basic,  underlying  conditions  will  continue  to  be 
in  force  for  the  prediction  period. 

o Short-term  forecasts  made  by  these  methods  usually  should  be  combined 
with,  or  weighed  against,  forecasts  made  independently  by  other  meth- 
ods before  a final  forecast  is  determined.  Other  statistical  tech- 
niques may  be  utilized  in  making  complementary  independent  forecasts. 
In  brief,  it  is  dangerous  to  use  regression  analysis  for  forecasting 
whenever  the  value  of  the  independent  variable(s)  falls  considerably 
outside  the  range  of  prior  data.  Regression  analysis  is  a very  useful 
tool  if  used  within  the  bounds  of  its  theory. 

o The  greatest  single  hazard  in  forecasting  is  that  events  that  were  un- 
foreseen and  unpredicted  by  the  forecaster  will  occur  to  "overturn"  a 
forecast.  Thus,  the  possibility  of  serious  error  in  a forecast  in- 
creases, as  a rule,-  with  the  length  of  the  forecast.  A forecast  cov- 
ering a period  six  months  in  the  future  is  less  likely  to  be  in  error 
because  of  the  impact  of  unforeseen  events  than  a comparable  forecast 
projected  five  years  ahead.  Similarly,  forecasts  made  during  periods 
of  relative  stability  are  more  likely  to  be  accurate  than  forecasts 
made  during  periods  when  unknown  and  unpredictable  conditions  may  be 


encountered 
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o The  techniques  of  time-series  analysis  examined  in  this  report  have 
proved  to  be  useful  tools  for  forecasting.  It  should  be  recognized  at 
the  start,  however,  that  these  techniques  cannot  be  applied  mechanist- 
ically; rather,  subjective  evaluations  must  play  a key  role  in  their 
use.  There  are  so  many  possible  models  from  which  to  choose  that  care 
must  be  exercised  to  prevent  unnecessary  and  uncalled  for  detail. 

There  are  several  reasons  why  subjective  considerations  are  important 
in  time-series  analysis.  First,  a generally  satisfactory  probability 
approach  to  such  analysis  has  not  yet  been  found.  Second,  even  if  a 
suitable  probability  approach  to  time-series  analysis  were  available, 
some  purely  subjective  evaluations  would  be  necessary  in  making  fore- 
casts. Whenever  we  examine  the  past  to  obtain  clues  about  the  future 
(as  in  the  case  when  we  use  time-series  analysis  as  an  aid  to  fore- 
casting) the  past  is  relevant  only  to  the  extent  that  causal  condi- 

* 

tions  previously  in  effect  continue  to  hold  in  the  period  ahead. 

Causal  conditions  seldom  remain  constant  from  period  to  period,  but 
rather  tend  to  be  constantly  shifting.  Thus,  the  connection  between 
the  past,  present,  and  the  future  must  be  continually  evaluated. 

E.  Sensitivity  Analysis  of  DOE  Forecasts  of  Midterm  Oil  and  Gas  Supply  for 
the  1978  Annual  Report  to  Congress^ 

The  objective  of  this  investigation  was  to  assess  the  quality  and  usefulness 
of  the  midterm  projections  of  oil  and  gas  production  given  in  Volume  III  of 
EIA’s  1978  Annual  Report  to  Congress  (ARC).  The  methodology  to  be  used  was  to 
be  based  on  the  knowledge,  methods,  and  results  from  NBS’  previous  validation 


*See  reference  [7]. 
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work.  The  study  design  included  a set  of  statistical  experiments  to  analyze 
the  sensitivity  of  MOGSM  outputs  to  systematic  variations  of  inputs,  and  a re- 
view of  some  of  the  MOGSM  methodology  and  its  impact  on  results,  possibly  in 
comparison  with  other  published  forecasts. 

Specifically,  our  statistical  experiments  included:  (1)  reestimation  of  find- 

ing rates*  by  various  statistical  methods  applied  to  historical  data  on  (re- 
gional) exploratory  drilling  and  the  corresponding  oil  and  gas  reserve  addi- 
tions; (2)  analysis  of  the  effects  of  Monte  Carlo  variations  of  regional 
resource  base  estimates  on  finding  rates  and  on  MOGSM  results;  and  (3)  a 
"response  surface"  analysis  (described  below)  to  identify  those  input  data 
elements  whose  variations  have  the  greatest  impact  on  MOGSM  outputs.  These 
experiments  were  to  be  supplemented  by  a Monte  Carlo  analysis  of  the  effects 
on  MOGSM  output  of  variation  in  the  critical  variables  identified  in  the 
response  surface  experiment,  but  this  step  was  aborted  owing  to  resource  limi- 
tations. Briefly,  we  found  that: 

o In  Experiment  1,  different  methods  of  estimation  yield  substantially 
different  values  of  regional  oil  and  gas  finding  rates.  When  applied 
to  MOGSM,  the  various  finding  rate  estimates  in  turn  lead  to  forcasts 
of  a wide  range  of  oil  and  gas  production,  as  shown  in  the  following 
table. 

Regression  Regression 

Method  of  Determination  Analysis  Time  Series  Analysis- 

of  Finding  Rate: *78  ARC  Met  Analysis  Alter.  Meth. 


Crude  Oil  (MM  Bbl/day) 

6,905 

6,510 

5,520 

Natural  Gas  (TCF/yr) 

14,476 

15,599 

10,311 

NGL  (MMBbl/day) 

1,152 

1,149 

840 

(average  values  over  the 

1985-2000  period) 

*As  used  in  the  model,  the  finding  rate  is  the  amount  of  proved  reserves  added 
per  foot  of  exploratory  drilling. 
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o Statistical  considerations  do  not  indicate  a preferred  estimation 
method  or  corresponding  (set  of)  production  forecast(s)  in  this 
instance.  The  finding  rate  estimates  actually  employed  in  generating 
the  1978  ARC  forecasts  appear  to  be  as  reasonable  as  any  of  the  alter- 
natives. But,  the  associated  uncertainty  is  high,  as  the  above  ranges 
suggest. 

o In  Experiment  2,  treating  the  undiscovered  regional  resource  bases 
(oil  and  gas)  as  random  variables  via  the  Monte  Carlo  analysis  leads 
to  uniformly  and  significantly  higher  oil  and  gas  production  forecasts 
from  MOGSM  than  does  the  current  (deterministic)  procedure.  Current- 
ly, MOGSM  operates  with  specific  regional  resource  base  estimates. 

For  example,  the  Series  C forecast  in  the  1978  ARC  is  based  upon  the 
50^  percentile  or  median  values  in  the  resource  base  estimates  set 
forth  in  USGS  Circular  725.  The  Monte  Carlo  analysis  instead  treated 
resource  base  estimates  as  log-normally  distributed  random  variables 
(consistent  with  USGS  Circular  725).  The  set  of  MOGSM-generated  oil 
and  gas  production  forecasts  resulting  from  the  analysis  shows  these 
properties,  relative  to  the  ARC  forecasts: 

- The  medians  of  the  Monte  Carlo  forecasts  are  all  markedly  more 
"optimistic"  than  the  1978  ARC  Series  C medium  geology  forecast, 
as  shown  in  the  following  table. 
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RESOURCE  BASE  PRODUCTION 

ESTIMATES 


1985 

1990 

1995 

Crude  Oil  (MM/Bbl/day) 

ARC  Series  C 

6,903 

7,081 

7,013 

Monte  Carlo  Median 

7,739 

8,819 

9,338 

Natural  Gas  (TCF/yr) 

ARC  Series  C 

16,777 

15,595 

11,802 

Monte  Carlo  Median 

21,754 

22,154 

19,828 

NGL  (MM/Bbl/day) 
ARC  Series  C 

1,338 

1,218 

1,092 

Monte  Carlo  Median 

1,586 

1,686 

1,539 

- The  range  of  the  Monte  Carlo  forecast  production  levels  is  uniformly 
higher  than  the  ARC  forecast  production  levels.  For  example,  the 
median  Monte  Carlo  crude  oil  forecast  is  comparable  with  the  highest 
ARC  forecast. 

- The  spread  between  the  5C^  percentile  and  95tn  percentile  of  the  set 
of  Monte  Carlo  forecasts  is  less  than  the  difference  between  the 
median  of  the  Monte  Carlo  forecasts  and  the  ARC  Series  C forecast. 
That  is,  the  MOGSM  forecasts  of  oil  and  gas  production  are  less 
sensitive  to  the  inherent  randomness  in  regional  resource  bases  than 
they  are  to  the  analytical  methods  by  which  resource  base  estimates 
are  applied  to  MOGSM. 


In  general.  Experiments  1 and  2 show  that  MOGSM  oil  and  gas  production  fore- 
casts are  highly  sensitive  both  to  inherent  variations  in  certain  MOGSM  input 
data  elements  and  (even  more)  to  the  analytical  procedures  used  in  defining  or 
estimating  these  elements. 
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Experiment  3,  response  surface  analysis'*-,  was  carried  out  in  two  stages. 
Variables  judged  to  be  of  strong  influence  were  divided  into  four  classes: 
geologic  (finding  rates),  economic  (price,  discount  rate,  various  costs),  geo- 
logic/technological (decline  rate,  recovery  factors),  and  drilling/operating 
(lease  acreage,  equipment  planning,  and  attrition).  These  furnished  a basis 
for  four  factorial  design  "subexperiments"  on  two  values  of  the  variables 
("high"  and  "low",  with  central  or  "base"  values  omitted),  each  of  which 
yielded  a linear  response  model.  Then,  variables  from  all  the  classes  showing 
the  greatest  influence  in  the  first  stage,  were  combined  to  construct  an 
"integrated"  response  surface. 

All  response  surface  equations  emerging  from  the  first  stage  experiments  show 

high  R“  values  (>0.90  in  all  cases)  indicating  that  the  model  is  well- 

approximated  by  these  linear  forms.  By  way  of  verification,  the  equations  all 

closely  fit  the  corresponding  center  points  (which  were  not  used  in  deriving 

the  equations,  but  reserved  for  calibration  purposes).  However,  the  equations 

o 

which  define  the  integrated  response  surface  do  not  show  particularly  high  Re- 
values (oil:  .88,  gas:  .64),  suggesting  that  the  true  response  surface 
requires  a nonlinear  approximation.  Time  did  not  permit  us  to  undertake  an 
investigation  into  possible  nonlinear  forms. 


^-Response  surface  analysis  involves  development  of  a simplified  model  of  a 
system  relating  its  inputs  to  its  outputs.  Until  recently,  the  technique  was 
applied  to  physical  experiments.  Here,  it  has  been  useful  for  sensitivity 
analysis  of  complicated,  large-scale,  mathematical  models. 
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Perhaps  the  dost  striking  aspect  of  this  analysis  is  the  prominence  of  "eco- 
nomic” data  elements  in  the  set  of  critical  elements.  By  contrast,  previous 
discussion  of  MOGSM’s  sensitivity  to  input  data  (EIA  1978)  focused  on  drilling 
and  geology-related  data  and  ignored  economic  data.  Similarly,  the  1978  ARC 
itself  emphasized  possible  changes  in  physical  phenomena  (e.g.,  the  "high  ge- 
ology" and  "low  geology"  scenarios)  more  than  possible  changes  in  prices  and 
costs. 

In  summation,  two  sets  of  input  data  elements  are  critical  to  MO GSM ’ s fore- 
casts, on  the  basis  of  this  experiment: 

- the  regional  resource  realizations,  and  through  them,  the 
regional  finding  rates,  and 

- the  economic  elements  appearing  in  the  table  below. 

We  have  not  had  the  opportunity  to  investigate  the  relative  importance  of 
these  two  classes  of  data,  but  such  an  investigation  involves  only  a modest 
extension  of  the  present  study,  and  would  be  most  useful  If  performed. 
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SENSITIVITY  OF  OIL  AND  GAS  PRODUCT  MEASURES  TO  CHANGES 
IN  SELECTED  INPUT  DATA  ELEMENTS 
(Integrating  Response  Surface) 


LEADS  TO  THESE  CHANGES  IN 
OIL  PRODUCT  MEASURE  & GAS 


FOR  THIS  MOGSMS 
INPUT  ELEMENT  

THIS  AMOUNT  OF 
CHANGE  

PRODUCT  ME AS  URE 
(in  103  Bbl/day) 

(in  103  BC) 

Price 

$10/Dbi(1) 

$i/mcf(1) 

834.2 

129.6 

Discount  Rate 

5%(1) 

-315.4 

-808.2 

Royalty  Rate 

io%C1) 

-232.1 

-216.7 

Drilling  Costs 

20%(2) 

-262.3 

0 

Primary  Recovery  Factor 

10%(2) 

569.5 

0 

Total-to-Exploratory 
Drilling  Ratio 

10%(2) 

-152.3 

477.9 

Notes : 

(1)  These  are  changes  in 

numerical  units : 

"dollars" 

or  "percentage  points". 

These  are  percentage  changes  from  nominal  or  base  values. 


(2) 
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Recommendations 

The  various  sensitivity  analysis  experiments  and  their  results  lead  to  these 
conclusions : 

o The  estimation  for  MOGSMS  of  regional  oil  and  gas  finding  rates  from 
historical  time  series  should,  in  the  future,  employ  time  series 
analysis  (second-order  exponential  smoothing  was  demonstrated  in  Ex- 
periment 1)  instead  of  regression  methods.  Time  series  analysis  was 
found  to  be  more  robust,  more  flexible  in  its  fitting  of  the  histori- 
cal data.  Moreover,  time  series  methods  (by  their  very  nature)  do  not 
require  independent  estimates  of  regional  resource  realizations,  which 
are  essential  for  the  present  regression  analysis  approach.  Indeed, 
the  time  series  analysis  itself  produces  independent  estimates  of  re- 
gional resource  realizations  (from  purely  statistical  treatment  of  the 
regional  time  series  on  reserve  additions  and  cumulative  drilling 
footage).  The  independent  estimates  obtained  in  this  experiment  cor- 
relate well  with  the  mean  regional  resource  estimates  set  forth  in 
USGS  725. 


Consequently,  MOGSMS  projections  can  be  decoupled  from  the  uncertain- 
ties in  regional  resource  estimates.  The  use  of  time  series  analysis 
in  this  application  removes  any  need  in  MOGSMS  for  regional  oil  and 
gas  resource  estimates  (which  are,  by  their  nature,  highly  uncertain) 
and  creates  an  independent,  statistically  based  source  of  estimates 
useful  for  other  EIA  purposes. 
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Another  property  of  time  series  analysis  further  recommends  its  use  in 
this  application.  Time  series  analysis  weighs  the  recent  years  more 
heavily  than  the  early  years  in  the  time  series.  Regression  analysis 
gives  equal  weight  to  all  of  the  years.  Because  time  series  analysis 
focuses  on  those  years  that  have  witnessed  the  greatest  changes  in 
technoeconomic  factors  and  are  likely  to  be  most  relevant  to  the  fu- 
ture, it  carries  not  only  greater  statistical  meaning  than  regression 
analysis,  but  also  greater  process  meaning.  (We  are  pleased  to  note 
that  EIA  has  adopted  time  series  analysis  for  the  1979  ARC  work  in- 
volving HOGSMS,  as  a consequence  of  this  study.) 

o The  analytic  procedures  demonstrated  in  this  study  lead  to  HOGSMS 
forecasts  of  midterm  natural  gas  production  that  are  significantly 
higher  (say  7 1/2  percent  to  15  percent)  than  those  shown  in  the  1978 
ARC  for  a given  set  of  assumptions  (e.g.,  the  Series  C scenario).  The 
reestimation  of  regional  finding  rates  via  time  series  analysis  (Ex- 
periment 1)  leads  to  an  increase  of  10.0  percent  in  the  MOGSMS  fore- 
cast of  average  gas  production  over  the  period  1985-1990-1995.  Simi- 
larly, the  treatment  of  uncertainty  in  regional  resource  base  esti- 
mates via  the  Monte  Carlo  analysis  (Experiment  2)  leads  an  increase  of 
7.7  percent  in  the  HOGSMS  forecast  of  gas  production. 
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The  regional  finding  rates  estimated  via  time  series  analysis  in  Ex- 
periment 1 are  themselves  subject  to  some  dispersion  or  uncertainty. 
One  could,  therefore,  carry  out  a Monte  Carlo  analysis  (similar  to 
that  in  Experiment  2)  to  explore  the  effect  of  that  variance  on  the 
MOGSMS  forecasts  of  gas  production.  Were  such  an  experiment  perform- 
ed, it  would  likely  yield  a gas  production  forecast  higher  than  ob- 
tained in  either  Experiment  1 or  Experiment  2.  (Indeed,  the  resulting 
increase  in  the  gas  production  forecast  might  approach  the  sum  of  the 
indicated  increases  in  Experiments  1 and  2.) 

A 10  percent  increase  in  projected  gas  production  over  the  midterm 
period  is  significant.  It  amounts  to  1.5  Quads/year  over  the  entire 
midterm  period — comparable,  for  example,  to  the  low  range  of  the  solar 
energy  contribution  forecast  for  1995-2000  in  the  recent  Domestic 
Policy  Review  on  Solar  Energy. 

o EIA  should  explore  the  prospects  and  benefits  of  a continuing  program 
to  characterize  uncertainty  and  treat  its  effects  in  various  DOE  anal- 
ysis systems.  This  study  can  be  considered  as  a pilot  demonstration 
of  some  of  the  practical  statistical  methods  that  can  be  employed  in 
an  operational  program.  It  also  illustrates  the  nature  of  the  results 
that  can  be  anticipated  from  such  a program.  In  this  study,  sensitiv- 
ity analysis  led  to  shifts  in  the  nominal  values  of  certain  forecasts, 
reassessments  of  the  "criticality"  of  certain  data  elements,  and  to 
sharpen  definition  of  the  associated  uncertainty.  For  example,  the 


28 


Monte  Carlo  analysis  reduced  by  a factor  of  10  the  variation  in  oil 
and  gas  production  projections  stemming  from  uncertainty  in  a key  in- 
put element  (regional  resource  estimates). 

o Uncertainty,  randomness,  or  measurement  errors  in  basic  data  need  not 
— a priori — preclude  the  use  of  quantitative  methods  for  policy  analy- 
sis. Good  analysis  has  a role  in  policy  making,  even  in  the  face  of 
"bad"  data.  For  example.  Experiment  2 showed  that  (within  the  frame- 
work of  the  MOGSMS  analysis)  the  inherent  uncertainty  in  the  ultimate 
magnitude  of  the  oil  and  gas  resource  base  does  not  lead  to  a corre- 
sponding range  of  uncertainty  in  forecasts  of  future  oil  and  gas  pro- 
duction. This  finding  resulted  from  confronting  and  treating  the  un- 
certainty in  the  data,  not  from  ignoring  it. 

F . Concepts  of  Model  Confidence^- 

One  of  the  goals  of  the  N3S  project  for  "Energy  Model  Validation  Procedure 
Development”  is  that  of  developing  methods  for  finding  the  degree  of  confi- 
dence in  results  obtained  from  different  analysis  systems  and  to  identify  the 
circumstances  under  which  the  systems  may  be  used.  The  concept  of  model  con- 
fidence is  not  clearly  defined  or  understood.  There  is  a tendency  to  equate 
model  confidence  with  model  validity,  as  validity  has  been  the  main  modeling 
activity  that  leads  to  the  acceptability  and  use  of  physically  based  models. 
However,  policy  analysis  models  and  their  use  in  decision  making  requires  an 
extended  definition  of  model  confidence  that  includes  validity  and  other 
measures  of  model  utility,  i.  e.  those  characteristics  of  a model  that  make  it 

*See  reference  [2]. 
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useful  and  usable  to  a decision  maker.  Thus,  the  main  effort  of  the  project 
was  an  attempt  to  define  model  confidence.  To  this  end  we  undertook  the 
following  activities: 

1.  Using  the  project  team’s  validation  and  assessment  experiences,  we 
developed  an  initial  set  of  confidence  criteria. 

2.  We  prepared  a discussion  paper  on  model  confidence  and  held  a work- 
shop to  (a)  define  model  confidence,  (b)  review  current  research  relevant  to 
the  concept  of  model  confidence,  (c)  discuss  a preliminary  methodology  to  be 
used  to  measure  confidence,  and  (d)  indicate  areas  of  future  research. 

3.  We  conducted  an  informal  survey  to  obtain  other  opinions  and/or  sig- 
nificant issues  related  to  model  confidence. 

Based  on  these  efforts,  we  proposed  the  following  initial  set  of  confidence 
criteria  for  a model. 

o Model  Definition — the  problem  and  model  environments.  The  information 
gathered  here  should  enable  the  decision  maker  to  determine  if  the 
problem  area  in  question  is  at  least  within  the  scope  of  the  model  pur- 
poses. 

o Model  Structure — the  theoretical  and  methodological  bases  of  the  model. 
The  information  gathered  here  should  enable  the  decision  maker  to  de- 
termine if  the  model  structure  has  limitations  that  preclude  its  use 
as  a decision  aid  for  the  problem  area  in  question. 
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o Model  Data — the  data  base,  data  sources,  and  procedures  for  data  trans- 
formations. The  information  gathered  here  should  enable  the  decision 
maker  to  determine  if  data  for  the  problem  area  in  question  are  avail- 
able at  reasonable  cost,  are  accurate  enough,  and  are  used  correctly 
by  the  model. 

o Computer  Model  (Program)  Verification — the  tests  and  procedures  used  to 
debug  the  subprograms  and  program,  and  how  the  consistency  between  the 
program  and  model's  mathematical  and  logical  description  was  estab- 
lished. The  information  gathered  here  should  enable  the  decision 
maker  to  determine  if  the  computer  program  is  reliable  and  if  it  ap- 
pears to  be  an  acceptable  representation  for  the  model, 
o Model  Validation — The  methods  with  which  the  computer  model  has  been 
analyzed  in  terms  of  its  ability  to  produce  results  that  can  be  relied 
upon  by  the  decision  maker.  The  information  gathered  here  should  ena- 
ble the  decision  maker  to  determine  that  the  model's  real-world  ap- 
proximation is  suitable  for  the  problem  area  in  question, 
o Model  Usability — The  resources,  procedures,  documentation,  accessi- 
bility, transferability,  and  maintenance  aspects  of  the  model.  The  in- 
formation gathered  here  should  enable  the  decision  maker  to  determine 
if  the  model  can  be  used  within  the  decision  maker's  problem  environ- 
ment. 

o Model  Demographics — the  model  antecedents,  orginators,  and  developers, 
past  users,  abstract,  cost,  and  current  developmental  activities. 

This  information  should  enable  the  decision  maker  to  determine  the 
model's  status  with  respect  to  past  achievements,  theoretical  and 
methodological  state-of-the-art,  and  the  expert  advice  that  went  into 


its  development. 
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Given  such  criteria,  the  key  problem  is  how  to  measure  the  levels  of  satisfac- 
tion for  each  criteria.  Our  approach  requires  the  decision  maker  to  ascribe 
one  of  five  distinct  levels  of  satisfaction  for  each  criterion.  Based  on  the 
information  gathered  on  the  criterion,  this  produces  a subjective  but  quanti- 
tative measure  of  the  personal  confidence  in  a model.  This  approach  has  not 
been  field  tested  as  yet. 

Recommendations 

1.  DOE  should  continue  work  in  model  confidence  by  sponsoring  a task 
that  develops  criteria  and  related  statements  from  the  perspective  of  DOE  and 
other  government  decision  makers. 

2.  A parallel  effort  should  experiment  with  the  organization  of  materi- 
als from  a DOE  model  assessment  project  into  sets  of  information  that  can  be 
used  by  a decision  maker  to  measure  the  seven  criteria  and  test  the  proposed 
confidence  methodology. 

G.  Conversion  Costs:  A Practical  Exercise  in  Model  Portability 

Experimental  attempts  to  set  up  and  run  the  Midterm  Oil  and  Gas  Supply  Model 
at  computer  installations  other  than  DOE's  370/168  were  planned  for  the  pur- 
pose of  development  of  a measure  of  computer  model  portability  and  for  testing 
the  MOGSM  operating  documentation.  The  initial  test  plan  specified  a facto- 
rial design  involving  various  computer  installations  and  two  levels  of 


*See  reference  [10]. 
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familiarity  with  the  new  computer  system  and  the  model  by  the  persons  respon- 
sible for  the  transfer.  Because  inadequacies  of  the  documentation  delayed 
completion  of  what  would  have  served  as  a partial  baseline  case  (operation  of 
the  model  by  the  NBS  assessment  team  on  the  DOE  "home”  computer),  only  one 
such  experiment  has  been  completed:  transfer  of  the  MOGSM  to  the  UNIVAC  1108 

at  NBS  by  a programmer/analyst  familiar  with  the  model  and  the  source  comput- 
er, but  unfamiliar  with  the  UNIVAC  1108  and  the  NBS  operating  system.  A suc- 
cessful transfer  was  made  using  the  base-case  scenario.  The  report  identifies 
various  hardware  and  operating  system-related  obstacles  to  portability,  pri- 
marily input/output  anomalies,  as  well  as  a previously  undetected  error  in  one 
of  the  model's  subroutines. 

Recommendations 

The  report  includes  several  recommendations  for  preparation  of  computer  pro- 
grams that  implement  large  scale  models,  intended  to  facilitate  transfer  of 
models  among  computer  environments.  The  two  most  general  of  these  are: 

o Use  ANSI  FORTRAN; 

o Make  the  system  configuration  as  simple  as  possible,  avoiding  large  num- 
bers of  interfaces  which  require  data  transfers  between  modules. 

The  report  provides  in  an  appendix  a tabulation  of  costs  incurred  in  dollars 

and  man  hours,  for  various  phases  of  the  transfer.  These  are  supplemented  by 

a description  of  the  contractor's  knowledge  of  the  model  and  the  two  computer 

systems  involved,  as  well  as  his  general  background,  to  furnish  a basis  for 

interpretation  of  these  costs. 
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Because  tlie  report  was  written  to  NBS,  it  is  directed  at  an  audience  presuma- 
bly expert  in  computer  models  in  general  and  in  the  UN1VAC  1108  computer  and 
operating  system  in  particular. 
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III.  ADDITIONAL  ACTIVITIES 

In  conjunction  with  the  basic  NBS/DOE  project,  NBS  organized  for  DOE  two  sym- 
posia. The  first.  Validation  and  Assessment  Issues  of  Energy  Models,  was  held 
on  January  10-11,  1979.  A proceedings  volume  was  published  with  that  title. 

It  can  be  ordered  from  the  Superintendent  of  Documents  as  NBS  Special  Publica- 
tion 569,  Saul  I.  Gass,  Editor,  Stock  No.  003-003-02155-5,  February  1980.  The 
material  in  this  560-page  volume  represents  a rather  complete  statement  of  the 
field  at  that  date. 

The  second  symposium,  Validation  and  Assessment  of  Energy  Models,  was  held  May 
19-21,  1980.  This  meeting  updated  the  state-of-the-art  knowledge  and  attempt- 
ed to  extend  such  knowledge  by  having  workshops  in  the  areas  of  (1)  validating 
composite  models,  (2)  the  measurement  of  model  confidence,  (3)  model  standards 
to  aid  assessment,  (4)  sensitivity  and  statistical  analysis  of  models,  and  (5) 
model  assessment  methodologies.  The  proceedings  of  this  conference  is  in 


preparation. 
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IV.  SUMMARY 


The  activities  summarized  in  this  report  are  not  to  be  construed  as  represent- 
ing a thorough  assessment  of  an  analysis  system.  That  was  not  the  purpose  of 
the  project.  Our  efforts  exemplify  approaches  for  addressing  various  aspects 
of  the  assessment  of  the  DOE  MOGSM  system  and  similar  models.  More  work  could 
be  done  under  each  of  the  reported  nativities  and  additional  activities*  could 
be  undertaken,  e.  g.  a comparison  of  model  results  to  known  outcomes.  In  ad- 
dition, there  are  other  approaches  which  could  be  taken  and  other  analysis 
systems  which  could  be  assessed.  Procedures  must  be  developed  that  translate 
the  vast  amount  of  information  gathered  during  an  assessment  effort  to  state- 
ments that  permit  a user  to  determine  if  the  results  produced  by  an  analysis 
system  can  be  utilized. 

It  is  recommended  that  DOE  and  other  sponsors,  developers,  and  users  of  analy- 
sis systems  initiate  and/or  continue  to  include  assessment  as  an  integral  part 
of  the  development  and  use  of  an  analysis  system.  It  is  no  longer  acceptable 
to  use  an  analysis  system  which  has  not  been  the  subject  of  assessment,  either 
by  the  modelers  themselves  (first  party  assessment)  or  by  an  independent  third 
party  assessment  team. 


- ■ 

We  were  especially  disappointed  that  project  time  and  funds  did  not  permit  a 
critical  analysis  of  the  structure  of  the  MOGSM.  We  feel  strongly  that  such 
an  analysis  would  be  most  beneficial  to  both  the  advancement  of  model 
assessment  methodology  and  the  improvement  of  the  MOGSM. 
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Although  it  may  be  argued  that  most  of  the  tasks  described  as  assessment 
activities  might  seem  to  be  those  that  should  be  done  in  the  course  of  model 
development,  i.  e.  merely  "good  modeling  practice,”  a more  careful  examination 
of  assessment  methodology  indicates  that  it  goes  beyond  that  characterization. 
For  some  areas  of  model  evaluation  no  methodological  procedures  exist  (e.  g. , 
determining  the  appropriateness  and  consistency  of  the  level  of  detail  and  ag- 
gregation of  a model);  while  for  others,  the  cost  of  using  current  methodology 
may  prove  prohibitive  (e.  g.,  performing  a sensitivity  analysis  of  all  input 
parameters  using  monte-carlo  techniques).  This  NBS  effort  and  other  similar 
efforts  are  early  steps  in  defining  and  advancing  model  assessment  method- 
ology. 

The  NBS  staff  will  concentrate  on  two  objectives  in  future  assessment  work. 

One  is  with  regard  to  the  "ex  post"  accuracy  of  model  results.  Here  we  intend 
to  define  and  develop  a measure  that  can  be  used  in  comparing  model  outputs  to 
historical  data.  The  second  area  is  an  extension  of  the  work  reported  in  [7] 
and  described  in  section  E.  We  intend  to  investigate  techniques  for  incorpo- 
rating correlations  among  input  variables  into  the  sensitivity  analysis  exper- 
iment. Throughout,  we  will  strive  to  achieve  a useful  level  of  communication 
with  the  community  of  non-technical  model  users. 
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This  last  issue,  communicating  with  the  non-technical  users,  is  an  important 
one.  At  the  Model  Validation  Symposium  held  at  the  National  Bureau  of  Stan- 
dards on  May  19-21,  1980,  this  issue  was  discussed  frequently.  While  it 
seemed  that  progress  has  been  made  in  identifying  valuable  techniques  in  model 
assessment,  not  much  effort  had  been  expended  in  responding  to  the  needs  and 
desires  of  policy  makers,  nor  in  learning  how  to  communicate  the  significance 
of  assessment  to  them. 

We  conclude  this  report  by  stating  that  even  as  the  development  of  model  as- 
sessment methodology  advances,  it  is  not  intended  to  lead  to  the  certification 
of  models  as  "valid."  It  has  (unfortunately  we  believe)  been  claimed  by  many 
in  the  energy  modeling  community  that  the  purpose  of  evaluation  is  certifica- 
tion. We  believe  that  more  attainable  and  useful  goals  for  model  evaluation 
are  to  enhance  system  understanding,  to  provide  directions  for  model  improve- 
ment, and  to  increase  credibility  and  confidence  in  energy  modeling  efforts. 
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